Repository navigation
feat(generator): clickable citations, parser and LLM arbitration - #120
Merged
Merged
Conversation
New :reflinks module that finds the citations written in the texts, e.g. (בראשית א, ד), (ב"ק כב:), (רמב"ם פ"ב מהל' שבת ה"ג), and resolves them to the cited line through line.heRef. Deterministic links (about 90%): explicit citations, a book or Bavli chapter name written before the parentheses, a folio inside a commentary on its tractate, and ibid (שם) within the same line. The ambiguous rest (שם across lines or bare, או"ח סי' without a book, מ"א/מ"ב, פ"X chapter or folio, Tanakh name before the parentheses) is arbitrated by an LLM over a closed list of candidates. Decisions are cached in generator/data/ref-arbitration.jsonl, keyed by a content hash of the case, so a rebuild never pays twice. The run fsyncs every paid answer, stops on no credit or a down server, honours a budget and the rate limits, and resumes where it stopped. ./gradlew :reflinks:arbitrateRefs only estimates the cost unless -Pconfirm is given.
- otzaria://inline-link anchors (path + 1-based line index): 19,296 resolved out of 19,296, all in שמירת שבת כהלכתה - מפתחות. - Sefaria refLink anchors: the English data-ref is translated through the titles and node names of the Sefaria export schemas (151/151). An anchor wins over the citation written inside it. - Bavli chapter names now come from the Bavli only: the Mishna and the Yerushalmi share them, which made every name ambiguous. A chapter name only resolves an explicit folio, so "(ב)" after it stays a note marker. - arbitrateRefs logs the deterministic links by method.
generateRefLinks scans a seforim.db, takes the deterministic links and the LLM decisions already cached, and writes them into a copy as <a class="ref" href="seforim://line/<id>"> around the citation, the Otzaria and Sefaria anchors being rewritten the same way. Links decided by the LLM carry data-llm="1". No network access; the source database is never modified. On the current corpus: 1,596,063 links in 800,404 lines; the visible text of the lines is unchanged. Also commits the decision cache (14,224 paid decisions).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for #111: make citations like
(בראשית א, ד)clickable. This PR adds the:reflinksgenerator module that finds the citations, decides their targets, and writes them into the text as<a class="ref" href="seforim://line/<id>">(בראשית א, ד)</a>(data-llm="1"when the LLM decided). The app side (rendering and opening the link) comes in a follow-up.How it works
The scanner reads every line, finds the
( … )/[ … ]groups and resolves them throughline.heRef.Deterministic (≈ 90 %, no LLM):
בביצה (דף יז)בפ' מרובה (דף עד)(chapter names from theChaptersalt TOC)(שם …)whose antecedent is in the same lineotzaria://inline-linkanchors (path + line index) and the 151 SefariarefLinks (Englishdata-reftranslated through the export schemas), all resolvedOn the current corpus: 1,588,668 deterministic links (explicit 1,145,154 · name before 315,228 · ibid in the line 56,346 · chapter name 33,090 · commentary's tractate 19,403 · Otzaria anchors 19,296 · Sefaria anchors 151).
Ambiguous (≈ 153k distinct cases), arbitrated by an LLM over a closed list of candidates:
שםacross lines or bare,או"ח סי'without a book (SA / Tur / MB / the commentary named before),מ"א/מ"ב(Kings or MA/MB),פ"Xafter a tractate (chapter or folio 80+X), Tanakh name before the parentheses.Measured on a held-out set of 420 annotated cases (Muse Spark 1.3, prompt v4, one prompt for ibid and one for references): 94.8 % accuracy, 98.5 % of the links it creates are right, 91 % recall. When unsure it answers "no link".
Decision cache
Decisions live in
generator/data/ref-arbitration.jsonl, keyed by a SHA-256 of what the answer depends on (context, candidates as title + heRef + snippet hash, prompt version, model). A rebuild reuses them without any network call; only new or changed passages need a new run. Targets are stored astitle|heRef, never as ids.ref-overrides.jsonl(manual corrections) always wins.Token safety (
:reflinks:arbitrateRefs)-Pconfirm; the budget ismin(-PmaxSpend, remaining credit − 0.25 USD)INVALIDand only retried with-PretryInvalid./gradlew :reflinks:arbitrateRefs -PllmKeyFile=~/key.txt -PmaxSpend=10 -PconfirmWriting the links
./gradlew :reflinks:generateRefLinkswrites the deterministic links and the cached decisions into a copy of the database (build/reflinks/seforim.db, the source is never modified), without network access. Links live in the HTML itself, so the app needs no offset mapping: its HTML parser only has to read thehref. On the current corpus: 1,596,063 links in 800,404 lines, visible text unchanged (checked on 3,000 random lines).Not in this PR
href, click → open the target line)Test plan
./gradlew :reflinks:jvmTest(31 tests): parser, Sefaria refs (real citations and the pitfalls found on the corpus), commentator mapping, store durability, and the arbitrator under 402, down server, budget, 429, transient errors, rejected and invalid replies